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ABSTRACT 


The purpose of this thesis is a discussion of developing human-like 
behavior in the computer. A theory of the human learming processes is 
first described. This leads to the presentation of a computer game 
which simulates the human capabilities of reasoning and learning. The 


program is required to make intelligent decisions based on past experi<« 


ences and critical analysis of the present situation. 
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I. INTRODUCTION 


In the past few years, much attention in the computer world has 

been given to the study and development of Artificial Intelligence. 

One goal of artificial intelligence is to develcp human-like qualities 

in the cimputer. One method of patterning human behavior is achieved by 
giving the computer the ability to (:) absorb data, (2) use inductive 
reasoning to make generalizaticns based on the data and abstract informa- 
tion from the data, (3) make decisions based on these abstractions and 
generalizations, and (4) learn from past experience. Admittedly this 
seems to be an enormous task, and in fact it is, yet this is how a child 
learns. To go one step further in modeling human behavior, and at the 
same time provide a reasonable limit on the size of the computer required, 
the ability to "forget" long-past experience could be developed in the 
system. 

The theory of the human learning process is discussed first in this 
thesis. The theory leads to the development of a computer program that 
learns based on data gathered from past experiences. The program uses 
its library of stored knowledge in making decisions. The example which 
is presented is a game named DIPLOMAT which simulates the representatives 
of two nations interacting in strategic negotiations regarding relative 
strengths, i.e., armaments. The game is played between two opponents, 
the computer and a person. 

The authors of the thesis consider that the interaction of the com- 
puter and a human in a decision-making environment is an excellent way 
of demonstrating the ability of the computer to develop human-like be- 


havior. 


Ii. THEORY 


One of the objectives of artificial intelligence ‘s to build a 
computer system that will effectively mimic the human learning process. 
This would he an enormous task if the system were to be so general in 
nature that it could handle any conceivable task, because it would in- 
volve representing the external world in a foin adaptable for the com- 
puter. When restricted to a small subset of the world, i.e., confined 
to one or two specific tasks, attempts at this objective have been fairly 
successful in that the systems do mimic and sometimes surpass the human 
in their performance in these areas. However when t:, ng to model the 
human brain in the general sense, the problem has beea found to be ex- 
tremely complex. 

A possible approach to the problem would be to model the human brain 
uging the physiological approach, i.e., to build the exact electrical 
network of sensors, storages, and connectors that form the physical makeup 
of the brain. Obviously this is impractical because the brain is com- 
posed of so many cells. Dr. R. L. Beurle, a noted English authority on 
artificial intelligence [Ref. 3] who has extensively studied the theory 
of brain models, estimates that the brain is composed of approximately 
10'° neurons (nerve cells). 

Another approach to the problem might be the psychological ap- 
proach. In this approach, it is necessary to model the logical structure 
of the brain rather than the physical structure. However in order to use 
this approach, some knowledge of the process of psychological development 


of human >Sehavior must be obtained. It would be best tc trace the learn- 


‘ ing processes from their beginning in a child, since a knowledge of 
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total human behavior usually requires almost a lifetime of study and 
experience. 

In the area of the development of the mental abilities of the child, 
the works uf one man stand out above all others - the works of Jean Piaget. 
Jean Piaget is a noted Swiss child psychologist who was educated at 
Hevchotel, Zurich, and the University of Paris. He has been a professor 
of child psychology and history of scientific thought at the University 
of Geneva since 1929, and is the director of the International Bureau of 
Education at the Institute J. J. Rousseau. Above all, he is noted for 
his research in the development of the mind from birth t» adolescence. 

The works of Jean Piaget, as presented in References 9 through 11, 
form the basis of this pee cues on the theory of the development of the 
human learning process. They are supported by References 15 and 16, 


which are basic references used in the American medical profession. 


A. THE LEARNING PROCESS OF THE HUMAN MIND 

Piaget contends tha’ the human mind consists of a finite number of 
structures, each consisting of a finite number of cells. Each of these 
cells contains an element of information which is a part of the human 
theught precess. He theorizes that there is a separate set of informa- 
tion structures for each function of the central nervous system which is 
composed of the brain and spinal cord. Many of these separate sets of 
structures are developed before birth, for example, a set contrcelling 
the action of the heart, another the function of breathing, a third 
controlling the flexing of the arms and legs. Piaget further contends 


that as relationships build, these structures are linked together to 


produce automatic reflexes and in general the structures are reordered 


and enlarged in successive phases. The central theme fer the human 
learning process is then the process of manipulation of the information 
structures, i.e., the building, storing, linking, rebuilding and re- 
linking of elements of information. 

It is easiest to describe the manipulating of these information 
structures by studying the development: of the brain of tne newborn baby. 
This is the approach presented by Piaget. At birth the cerebral cortex, 
which is composed principally of neurons, is largely undeveloped, and 
the infant is basically a reflex organism. The reflexes are the result 
of the manipulation of the information structures before birth. Other 
than the regular actions of the heart and other vitai systems, most muscle 
activity is random, lacking direction from the brain. 

One of the first learning processes of the infant is the learning 
of spatial relationships. While flexing his arms and legs, he touches 
the side of the crib. Relationships are built in his mind between the 
infonaation structures for the sense of touch and those of muscle control, 
and a new structure built up vagarding an awareness of the confining walls 
of the crib. These structures become libraries of information. Ocher 
libraries build up, for example he grows to associate his mother with 
warmth, comfort, and nourishment. The building of relationships between 
the structures, and the reordering of the structures and links, formes an 
associative memory within the brain. 

It is important in the stucy of the humar brain to understand that 
these activities are the result of relationships between the libraries 
even though the information in their structures is dissimilar. Once the 


links are established, the dissimilar sets function together. The act of 


crying which is the result of combining the structures of muscle control, 


an awareness of a specific need, respiration control, and others is a good 
example. The operation that is accompiished constitutes some action of 
uniting or separating, placing cr displacing, airanging or disarranging 
elements of information into sets of structures. As the relationships 
develop within the brain, the mind learns how to use the data which has 
been structured within its memory to produce a desired result, such as 


rolling over or sitting up. 


B. REPRESENTATION OF THE HUMAN LEARNING PROCESS 

The process of thought is the result of the human becoming aware of 
the relationships that link ce’ structures in the associative memory of 
his brain. Piaget explains the mechar ‘sm of thought as a movement which 
evolves when an aravticay of the relationships becomes sufficiently 
advanced to permit the individual to combine the information from several 
structures into a single idea. The resulting thought may cause other re- 
actions such as body movement, and thus may cause new structures to be 
created or new Links to be built. The actual process involved in the 

- human brain in conceiving a thcught is not definitely known, and must be 
extremely complicated. If such a scheme is to oe computerized, however, 
it must be made deterministic. 

In an attempt to make the process a deterministic one, it is neces- 
sary to summarize some facts deduced from the discussion of Piaget's 
works. One result of the thought process is the acquiring of an idea, 
which is the outcome of the interaction of cells or structures. ‘The 
combination of old structures in a new way may lead to new and possibly 
improved concepts. The new relationships may be formed by applying de- 


ductive reasoning to some information extracted from the interconnected 


~ structures. In this wey, the learning of a new concept may be achieved. 
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Thus learning is not merely an additive process, i.e., the piling of 
one disjoint piece of information atop another and another and another. 

The number of cells or structures is not the criteria for learning, 
rather it is the effective combining of the stored information that re- 
sults in intelligence. Once a certain puint is reached in the process 
of hum-n development, the physical size of the brain does not rapidly 
become larger and larger, rather information is restructured, old info~na- 
tion may be fe~gotten and new information stored in its place, and new 
links established. 

These two techniques, structuring of n. ory and learning by inter- 
action of the structured information, are utilized in the structuring of 
a computer memory and the development of a program which will exhibit de- 
ductive reasoning based on learning. This is the example presented in 
this thesis. Considerable research and soul searching went into formu- 
lating a worthwhile application for the model of human intelligence. 

The application had to be generel enough to apply to real-life situa- 


tions, yet not overly complicated. 


C. BACKGROUND OF THE GAME OF DIPLOMAT 

Several recent periodicals have been devoting considerable attention 
to games as an application of artificial intelligence. One game in parti- 
cular hag been the subject of much of the attention, the game know as 
the Prisoner's Dilemma. The classic prisoner's dilemma is described in 
reference 7 as follows: 

“Two suspects are taken into custody and separated. The 
district attorney is certain that they are guilty of a specific 
crime, but he does not have adequate evidence to convict them 
at a trial. He points out to each prisoner that each has two 


alternatives: to confess to the crime the police are sure they 
have done, or not to confess. If they both do not confess, then 


the district attorney states he will book them on some minor 

trumped-up charge such as petty larcency and illegal possession 

of a weapon, and they will both receive minor punishment; if 

they both confess they will be prosecuted, but he will recom- 

mend less than the most severe sentence; but if one confesses 

and the other does not, then the confessor will receive lenient 

treatment for turning state's evidence, whereas the latter will 

get ‘the book’ slapped at him." 
One reason the prisoner's dilemma has been discussed so much is thac there 
are numerous situations in the world that have some of the characteristics 
of this game (or extensions from this game such as the addition of parti- 
cipants and/or strategies). Most economic situations that require a 
choice among a finite number of strategies have these characteristics. 
Consider, for example, gasoline service stations located close to one an- 
other, each of which can lower its prices. Regardless of the price one's 
competitors set, any one manager is better otf, in the short run at least, 
cutting his price. If all cut prices, however, the total volume of bus- 
iness is the same as if none cut prices, but the total revenue is less. 
On a larger scale, consider wheat farmers in a country without govern- 
mental price and production coacnalie. Any one farmer is better off pro- 
ducing wheat as long as his marginal cost is not greater than the price. 
He will be able to sell all he can produce at the going market rate with- 
ouc affecting the price. If all farmers produce maximum amounts, however, 
the price will be pushed down and all will be worse off than if each had 
restricted his production. On the worldwide scale, there is the problem 
of disarmament. One country can be more powerful (or secure) by arming, 
but nothing is gained if all arm. All countries would be better off if 
all disarmed in that the money not spent on defnese could be spent for, 
say, consumer goods or for correcting social ills. 


More interesting than the one-time classic prisoner's dilemma is the 


iterated game, i.e., a game composed of many moves. In the overall 
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picture of the iterated game, each player must (in general) forsake the 
possibility of maximizing his own short-run profit to enjoy the greatest 
payoff by maximizing his long-run profit. With a one-trial game and an 
unknown rival, it is difficult to imagine the wisdom of choosing a medium- 
gain, little-risk, cooperating type strategy, when more can be gained (or 
lost) by choosing the high-risk high-payoff strategy (nothing ventured, 
nothing gained). The single trial stecarion eliminates both the possibil- 
ity of future cooperation and the possibility of punishing a rival for non- 
cooperative action in one trial. Dr. Lester B. Lave of the Carnegie 
Institute of Technology [Ref. 6) has studied factors affecting cooperation 
in prisoner's dilemma type games. He has found that the single-trial game 
and the multi-trial game are basically equivalent in the formal sense in 
that the expected values of the two games have the same range across differ- 
ent groups of opponents. However, the games are quite different with re- 
spect to negotiating cooperation among different participants. The ex- 
pected values of the two games are not equal for a given rival, since 
certain forms of behavior can induce cooperation or competition. He based 
these results on experiments conducted using human competitors only, and 
did not introduce computer gaming into his research. He further found 
through experimentation that when a game was iterated, it was possible 
to display behavior that induces or stifles cooperation. He also found 
that it was possible for the players to develop comnunication between 
them using the choices in the game, but that this rudimentary form of 
communication took time to establish and function. He found that the 
longer the game, the more likely it was that a stable cooperative solu- 
tion could be achieved. 

Other experiments with a complex decision task showed that experi- 


ence from previous tasks was a large factor in success. The conclusions 
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drawn from studies conducced at the Human Performance Center, Ohio State 
University, [Ref. 13) were that after gaining experience when tested under 
realistic circumstances, the experienced subjects were in general less 
conservative than naive subjects who received no such prior training. 
They were willing to take more risks to achieve higher overall gains. 

The study of concept attainment by the machine has also been the 
subject of study. In order for a human to vias a game of this nature, 
he must be able to form a few concepts of the game itself and of his 
opponent. It can be likened to the game of poker in that opponents must 
asdiee the type of individuals playing the game. Dr. Frank B. Baker, a 
professor of educational psychology at the University of Wisconsin, has 
studied the theory of concept attainment and developed a computer program 
which demonstrated the theory in a simple decision task requiring the 
identification of common attributes among different sets [Ref. 1}. In 
Reference 2, Dr. Baker states: 

"If computer programs are to serve as useful models of 

congnitive behavior, their creators cannot avoid coming to 

grips with the necessity for establishing an internal organi- 

zation for their model which implements the higher level 

cognitive behavior associated with the human capacity for 

self-direction, autocriticism, and adaptation." 
In computer game playing, the concept of the game itself is built into 
the game, however the concept of different methods of play or types of 
strategies that the computer may face is something that must be attained 
as it plays. 

Most of the behavioral tests conducted to date have been between 
humans. However, in the past few years, more attention has neen devoted 
to simulating these tests on the computer using interaction between man 


and the machine. Professor Roman J. Weil of the University of Chicago 


stated in Reference 18 the advantage of using the computer for this appii- 


cation quite well: 


"The ruilosophy underlying the computer approach is this: 

If a program can be constructed that, when placed in a prisoner's 

dilemma situation, exhibits behavior like the behavior of people 

when placed in the same situation, then that program will be a 

powerful tool for generalizations." 

Professor Weil goes on to say that if the computer can be made to 
simulate the human in organizing data and making decisions under all 
simuiated conditions cf risk and stress, it will be possible to accunu- 
late more data and more accurately predicc human behavior in the same 
environment. 

The game of DIPLOMAT presented in this thesis is basically an ex- 
tended version of the prisoner's dilemma. It incorporates an iterated 
game technique with the gave lasting anywhere from ten to fifcy moves, 
and two opponents choosing from among three strategies. Additional compli- 
cations to the prisoner's dilemma basis are inserted by varying the pay- 
offs to the participants as a function of previous moves, and by inserting 
an unknown random variance into the payoff table. 

In order to be successful in this game, the participants must per- 
form most of the tasks listed in the introduction as a goal of artificel 
intelligence, in addition to performing the task of concept attainment. 

In particular, the computer must analyze the situation of the game at 

the time of the move and refer to past games and past moves in the pre- 
sent game to determine its opponent's probable strategy. It must then 
abstract enough information from his prior and present knowledge to select 
a strategy, and correctly analyze the results of the move in order to 
store (remember) meaningful experience. In addition, it must form con- 
cepts regarding the reliability or honesty of its oppo:ents. The parti- 


cipants in this game may or may not be completely truthful in their 
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negotiations, which is certainly characteristic of actuai diplomats at 
the conference table. The computer, then, must form estimates of its 
opponent's reliabiiity and factor this into its selection of a strategy. 
In the cirst game played, the computer has no prior knowledge upon 
which to draw, and so must reason from an analysis of the present situa- 
tion. With each move however, the system acquires more experience and 
thus has a better base from which to draw in selecting strategies. When 
the computer plays its second game, it has the experience of the first, 
with one winning and one losing strategy, to use for reference. In 
general, the more games it has played, the more experienced it is and 


the better it performs in making important decisions. 
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III, THE GAME 


DIPLOMAT models the representatives of two nations interacting in 
strategic negotiations regarding armaments. It is basically a non-zero- 
sum two person rectangular game, using the phraseology of formal game 
theory. Each nation is given three choices of possible strategies re- 
garding armaments: 

Strategy 1: Increase Armaments (Arm) 

Strategy 2: Maintain the Status Quo 

Strategy 3: Decrease Armaments (Disarm) 
The following basic concepts govern the decision of the strategy to be 
followed by each side: 

Each nation starts the game with zero strength and zero wealth, 
where "zero" implies a deviation from the average rather than absolute 
zero. 

Arming increases strength by one unit, disarming decreases strength 
by one unit, and maintaining the status quo cloes not change strength. 
Arming costs money decreasing wealth, disarming gains wealth, and main- 
taining the status quo may increase or decrease wealth, depending on the 
strength of the player: If the player is strong in armaments, it will 
cost him more for upkeep and maintenance and hence decrease wealth; if 
the player is weak, armament upkeep is low and maintaining the status 
quo should gain some wealth. The basic changes in wealth for the differ- 
ent strategies are shown as a function cf strength of the player in 


Table I. 
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Strength th p Severe Sener ere eres 
§ < -6 0 3 3 

6 2S < 4 0 2 3 

4 4S & -2 -1 2 3 

-2 <S < +2 -2 1 4 

42259 £44 <3 0 3 

4 <S$ 24 “4 -2 3 
Ss > +6 -6 -3 2 

Table I 


Wealth Changes Versus Strategies As 
a Function of Player Strength 
Changes to the basic values in Table I are generated by a random 
integer amount between the values of -2 and +2. These changes, which 
are to simulate economic conditions, are generated at random times during 
the game; hence, a given economic condition may last for only one move 
or for many moves. The economic condition for each player may be differ- 
ent, as each opponent uses a different random number generated from the 
same random number seed. The economic condition in effect for each 
player is not furnished to either opponent, but must be estimated based 
on the results of each move. The economic conditions of -2 and -1 mimic 
those times when prices are high, and the conditions of +1 and +2 mimic 
those times when the cost of living is relatively low. These values are 
added to the basic wealth change values of Table I to determine the 
actual wealth changes for the strategies chosen. 
Point values for each participant are determined after each move 


according to the formula: 


POINTS = (Previous) + 2 X (Wealth) + 5 X (Strength) 
Points Change Change 


Bonus points are given for the following combinations of strategies: 
If both opponents cooperate in disarming, each receives two 
bonus points as a reward for their cooperation. 
If both opponents arm, each receives -1 bonus point, because 
both have spent money without acquiring eny relative advantage. 
If one side arms and the other disarms, the opponent who arms 


"outfoxing™ the other. 


is awarded four bonus points for 
The length of the game is at least ten and at most fifty moves. 
Between these values, a random selection is used for the decision to end 
the game; as the number of moves increases, the greater the chance of 
random termination. Experience has indicated that the average length of 
the game is twenty moves. 
The winner of the game is decided by one of three methods: 
Normal Termination: The participant with the most POINTS at 
the end of the game is declared the winner. 
kbnoveal Termination: : The game may be abnormally terminated, 
even before ten moves have been completed, in two ways. The game is 
stopped if one nation's wealth becomes thirty units greater than the 
other, and the richer nation is declared the winner. Similarly, if one 
participant becomes stronger in armaments than the other by ten units, 
the game is stopped and the stronger nation is declazed the winner. 
In playing the game of DIPLOMAT, each side initially declares a 
proposed strategy called that participant's Concession Point, with each 
side taking turns declaring the first concession point. The proposal of 


each player is used as an aid in deciding that player's probable actual 


strategy. After both ccncession points have been declared, the computer's 


move is locked into the system and the human participant is asked to 
declare his final strategy for that move. Of course, the concession 
point and the strategy need not be the same, but wisdom must be used in 
selecting proposals versus actual strategies in order to maintain a high 
depree of reliability, yet keep the opponent off guard as to the actual 
strategy to be chosen. 

The program is written es a main routine which controls the running 
of the game itself, and forty-one subroutines. Seven of the subroutines 
assist in controlling the game and in performing list-processing chores, 
nine of the subroutines assist the program in accomplishing its reasoning 
capabilities, and the other twenty-five subroutines are necessary for 
accomplishing the task of learning. Among the tasks performed by these 
forty-one subroutines are setting up the memory, saving and restoring 
experience, selecting a strategy and a concession point, determining the 
opponent's reliability, pattern matching data from previous games and 
moves to take advantage of prior experience, and so forth. The program 
is written in the FORTRAN IV language, and is designed to operate on- 
line on a computer terminal using the Cambridge Monitor System. A 
complete listing of the program is contained in Appendix C. 

Figure 1 is a macro-flowchart of the game. DIPLOMAT progresses in 
the following manner: 


After initializing counters, point values, and payoff tables, 


the system sets up the memory cells in an associative memory structure, 
filling in the data from previous games. The initial economic condi- 
tions are also detcrmined. 

The player (hereafter called SOUTH) signs into the system 


with his name. The computer program (hereafter called NORTH) then 


INITIALI- 
ZATION 


; REBUILD Ex- | 
PERIENCE €¢ 
INPUT 
PLAYER DATA 


) Ce) 


Comeurer Plavar 
COMR SELECT PLAYER 
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COMPUTER COMPUTER 
DECLARE SELECT 
CONCESS, PT. - STRATEGY 
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DE CLARE 
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STRATEGY 
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* NO END o YES 
SAME 
? 
: GAIN EXPERI- 
SHIFT ENCE AND 
A TURN B 


DECLARE 
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. Figure 1. Macro Flowchart of the Game DIPLOMAT 
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checks to see if he has played this opponent before, and it so brings 
his history to the top of the catalog of players. 

The player inputs a random integer number for use as a seed 
in generating random numbers throughout the program. The computer simu- 
lates flipping a coin to see who will make the first proposal during the 
first move. SOUTH calls the flip of the coin. (In the discussion which 
follows, it will be assumed that SOUTH won the toss). 

Having won the toss of the coin, SOUTH must make the firs¢e 
concession point. To aid him in making his selection of a proposal and 
ultimately of a strategy, a payoff matrix is presented to him showing 
the approximate payoffs to SOUTH (relative to NORTH) for the various 
combinations of strategies. Approximate payoffs are shown rather than 
actual because neither SOUTH nor NORTH have access to the existing 
economic conditions. A sample payoff matrix as presented to SOUTH is 


shown in Figure 2. 


SOUTH 
STRATEGY Jodi dicicdicicick biieiciceicieictbicteick 
* 
N 1 * el] 1 2 
0 * 
R 2 * l 0 1 
T * 
H 3 * 2 o} 2 
Figure 2 


Approximate Payoff Matrix Presented to 
Player (SQUTH) 


The approximate payoff matrix changes according to the strengths of 


NORTH and SCUTH because of the differences in wealth for the various 


strategies as shown in Table I. 


seer Te Ee ee yee a ee ee 


SOUTH may propose a concession point of 1 (arming), 2 (main- 
taining the status quo), or 3 (disarming). NORTH then considers SOUTH's 
concession point, attempts to determine if SOUTH is being honest, analyzes 
his payoff matrix, and responds to SOUTH with his concession point. 

NORTH also decides upon a strategy at this time. 

SOUTH considers NORTH's concession point and decides upon a 
strategy for the move. Both sides then enter their strategies into the 
system and the results are tabulated and presented for analysis by both 
participants. 

For the next move, NORTH will declare his concession point 
first. The game proceeds in this manner, alternating between NORTH and 
SOULH as to who is first to declare a concession point. After each move, 
each side analyzes the results in order to determine the economic condi- 
tions in effect. After ten moves have been completed and after each move 
thereafter, a random number is generated and tested to deternine if the 
game should end. 

At the end of each move, changes to the payoff tables gener- 
ated as a result of changing strengths of the participants are calculated 
and inserted into the system, and it is determined if it is time to 
change the economic conditions. If so, they are calculated and inserted 
into the game. 

A sample output of the program as exhibited at the counter terninal 
is located in Appendix B. 

In playing this game, the computer program maintains data regarding 
past moves and past games in order to draw upon this experience in 


selecting strategies and concession points in future moves and in future 
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games. It thus forms concepts of each player as it proceeds. At the end 
of each move, some of the learning subroutines are called upon to update 
the short-term memory in order to store data for us’ in playing the game 
in progress and for maintaining running totals. At vhe end of the game, 
others of these subroutines calculate the game totals and determine the 
characteristics exhibited by both participants for inclusion in the long- 
term memory. This is the experience gained by the program from this game. 

The datz which forms the experience is kept in an associative memory 
structure mainly for ease of manipulation, but it is considered that this 
patterns the human in organizing data and experiences in his mind. In 
studies conducted of neural nets, it has been found that the human will 
take data, organize it into logical structures based on ‘determining re- 
lationships setween units of the data, and siore it in an associative 
net accordingly [Ref.s 2 and 3). 

Diegrams of the memory structure of DIPLOMAT are contained in Appen- 
dix A, and were conceived by the authors after critical analysis of the 
structure of data maintained about players during hand simulation of the 
game. 

Besides exhibiting the human attribute of learning by storing away 
past experiences, the program mimics the human in its reasoning cap- 
ability in deciding upon strategies and concession points. The method 
of reasoning was also patterned after analysis of the mental reasoning 
used during hand simulation of the game. These two attributes of 
reasoning and learning are discussed in greater detail in the sections 


which follow. 
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A. THE REASONING PROCESSES 
Two of the subroutines of the program are designed to mimic the 
reasoning processes of the human. These are the strategy decision sub- 
routine (STRTGY) and the concession point decision subroutine (CPNDCN). 
These subroutines in turn call on many other subroutines to determine 
optimum strategies, probable moves of the opponent, next moves of the 
computer, and so forth. Both the strategy and concession point decision 
subroutines were written based on the thought processes used by the 
authors in playing the game by hand. 
1. Subroutine STRIGY 

This subroutine-has three main functions. The first is that 
of strategy decision based on an analysis of all the factors available. 
This decision is final if the player's concession point is known, other- 
wise it is a tentative decision until the declaration of SOUTH's con- 
cession point. The second function is that of reconsideration of the 
tentative strategy after SOUTH has declared his concession point, and 
results in the final selection of a strategy. This function is called 
upon only if NORTH was first to declare a concession point. The third 
function is analysis of the completed move in order to determine if a 
better choice of strategy could have been made. If so, an adjustment 
of the calculations performed in the decision portion of the routine is 
accomplished. 

Macro-flowcharts of STRTGY are shown in Figures 3 and 4. The 
subroutine may be called upon either two or three times each move, de- 
pending upon who submits the first proposal. A series of flags are 


used to determine for which function the subroutine is being called. 


The algorithm for making this determination is shown in Figure 3. 


28 


CALL 
STRTSCY 


Anwacrsis 
WoT 
OS hep 
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RETURN 
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CONSIDER ALL 
OTHER. 
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in Figure 3. 


Macro Flowchart of Subroutine STRIGY indicating 
Determination of the Desired Function 
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Figure 4. Basic Algorithm of STRIGY 
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Figure 4 provides the basic algorithm of STRTGY. The decision 
upon a strategy for the move is based upon the consideration of four 
factors, each of which are weighted in a "polynomial" feshion. These 
four cerms of the polynomial are based upon: 

1)  SOUTH's concession point, 

2) POINTS relative to SOUTH, 

3)  Previcus experience, and 

4) Economic conditions. 
The coefficients assigned to each of these terms are inserted into the 
system at the start of the game, and may become modified as the game 
progresses, as discussed in the descrip: ‘on of the analysis function. 

Even before considering any of the four factors, the computer 
checks to ensure that it is not in danger of losing because of relative 
wealth or strength disadvantage. If it is, the system immediately selects 
as its strategy the one which will gain the most wealth or strength, as 
needed, and returns to the main program without considering any of the 
four factors. If the computer finds that this emergency action is neces- 
sury, later analysis of the move is not nactonned: 

If emergency action is not necessary, each of the four factors 
are considered and used in determining a final strategy for the move. 

The first factor to be considered is SOUTH's concession point. 
The computer first determines if it wants to call for a renegotiation. 
Renegotiatior, is requested if it finds that its opponent's proposel was 
to arm and that he is quite strong already; it requires SOUTH to submit 
a new concession point, although the new proposal may be the same as 
the original. The next step after considering (and possibly carrying 


out) renegotiation is to attempt to determine SOUTH's probable forth- 


- coming strategy. The computer selects candidates for its opponent's 


strategy based on weighted reliability estimates, patterns of dishonesty 
in SOUTH's proposals, history of previous games with this player, and 
formal game theory. After deciding upon SOUTH's probable strategy, 

the computer selects ag the first term of the polynomial the strategy 
which maximizes NORTH's gain relative to SCUTH if SOUTH actually selects 
that strategy. 

The second factor to be considered in selecting a strategy is 
based on NORTH's points relative to SOUTH. Depending upon whether the 
computer is behind, ahead, or even with its opponent, this term of the 
polynomial is set to the strategy which maximizes NORTH's nossible gain 
ee piniaiiet its possible loss. Formal gam theory is used in selecting 
these possible strategies. 

The third factor to be considered is the most difficult one, 
because it is based upon previous experience. If SOUTH has been a pre- 
vious opponent (within the last ten opponents), there exists a history 
of his previous games in the computer's long term memory. NORTH can use 
this information in predicting SOUTH's probable moves. In addition, 
NORTH can search other strategy types contained in its libraries of past 
games in order to select previously successful strategies to use against 
its opponent. After several moves have been completed, the computer 
searches all the strategy listings in its “experience” in order to classify 
SOUTH's general pattern of strategies. Each of these listings may con- 
tain pointers to other strategy patterns which have proven successful 
against SOUTH's pattern in the past. NORTH can also pattern-match its 
own strategy pattern against these existing in the library in order to 


determine its own predicted move. The computer occasionally selects a 
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strategy opposite te the predicted cre to avoid being stereotyped by 
SOUTH. This is called a "guess-opposite" selection of a strategy. It 

is the third factor of the polynomial hich gains from the learning 
capabilities of the machine. The method of learning and pattern-matching 
is discussed in greater detail in a later section. 

The fourth and final factor to be considered in selecting a 
strategy is based on an estimate of the economic condition in effect. 
After the completion of each move, the program analyzes the results to 
determine if they match the expected values. If not, it estimates the 
economic conditions and sets this term of the polynomial to the strategy 
which takes the most advantage of the state cf the economy. For example, 
4£ prices are low, it is. probably the best time to arm, but if prices are 
abnormally high it may be too expensive to arm at that time. 

After all four factors have been considered in selecting a 
strategy, the system conducts a vote to determine the choice for the move. 
If SOUTH's concession point has not been declared, the coefficient of the 
first term is set to zero, and a tentative strategy is chosen based on 
the other three terms. When it becomes time for the reconsideration, 
the vote is taken of all four terms for deciding the final strategy. 

The vote is accomplished by summing the coefficients of the temns voting 
for each of the three possible strategies. The strategy which receives 
the greatest sum is the "winning" strategy; in case of a tie, the com- 
puter selects from the tieing strategies the one which will result in 
the greatest absolute point gain. 

Having selected a2 strategy, control is returned to the main 
program which either calls upoa Subroutine CI.:DCN to decide NORTH's con- 
cession point, or else locks the strategy into the system awaiting SOUTH's 


indication of a strategy. 
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The third function of Subroutine STRTGY is that of analyzing 
the completed move in order to determine if a better strategy could have 
been chosen. The system does this by entering NORTH's relative payoff 
table with the actual strategy chosen by SOU.H «> determine the strategy 
which provides the maximum relative gain. If the computer determines 
that the correct strategy was chosen, no adjustments are performed and 
‘control is returned to the main program. If, however, the best strategy 
was not picked by NORTH, the computer determines if any of the votes cast 
for the different strategies matched the best possible strategy. If any 
are found, the coefficients corresponding to those terms are then in- 
creased (tuned up), and those assigned to the wrong terms tuned down. 

If no votes are found, the analysis was unsuccessful for that move. 

If the computer ultimately wins the geme, the final values of 
the coefficients for each of the terms are inserted into the long-term 
memory of the player for use as the initial values ir the succeeding game 
with that player. This is done because the system found these coeffi- 
cients successful and they would probably provide a better base from 
which to start the next time. This is but another part of the learning 
process of the system. 

Thus, Subroutine STRTGY attempts to determine a strategy tc be 
followed in the gaze of DIPLOMAT in much the same way that a human reasons 
through the game. Similar to a human, if the r-:asoning process results in 
a wrong answer, the system attempts to improve itself. 

2. Subroutine CPNDCN 

This is the second of the program's subroutines which employs 
human-like reasoning. Its purpose is to determine a concession point to 
be proposed to SOUTH after having selected a tentative (or final) strategy. 


Figure 5 provides a macro-flowchart of this subroutine. 
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CALL 
CPNDCN 


DECLARE 
ACTUAL 
STRATEGY 


DECLARE 
DiS ARM 


DECLARE 


STATUS QUO 
AS CONC. FT: 


OBTAIN NCRTHS 
PREHICTED 
MovVE 


2 Figure 5. Basic Algorithm of CPNDCN 


NORTH's concessicn point must be carefully chosen. In order 
to keep the player off guard, the computer must try to maintain at least 
a facade of reliability, yet not be so reliable as to be "read like a 
hook." If NORTH and SOUTH are each following a pattern of moves, the 
computer should indicate that it expects to follow the pattern, even 
if it has no such intention. 

For the initial moves, the computer uses a semi-random pattern 
of selecting proposals. Weight is given to choosing either the actual 
strategy proposed, or to choosing a passive (i.e., disarming or status 
quo) declaration. After the initial moves have been completed, however, 
the computer utilizes much of the data it has been accumulating on the 
player during the game. In addition to pattzrn-matching previcus moves, 
the computer maintains estimates of the player's reliability as well as 
its own. EN is NORTH's estimate of SOUTH's honesty, and ES is what NORTH 
thinks SOUTH estimates for the computer's reliability. These are both 
weighted values, wita the most weight given to the recent moves. EN and 
ES are calculated after each move by comparing the actual strategy with 
the concession point. The values of ™N and ES drop accordingly each time 
there is not a match. | 

If ES is high, is increasing, or at least is greater than EN, 
the system determines its predicted move from pattern-matching its own 
strategies, and uses a predicted move as its concession point. If, on 
the other hand, NORTH's reliability needs improving, the subroutine 
normally selects the actual strategy determined by STRTGY as the computer's 
concession point. 

Sie As with STRTGY, CPNDCN was written based on hand simulation of 


the game. Notes were kept on the reasons behind each decision, and 
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uscd to help determine the methods of reasoning developed in these 


suoroutines. 


B, MEMORY STRUCTURE AND THE LEARNING PROCESS 

The information within this section is somewhat more detailed be- 
cause it describes the attempts to exemplify the concepts of Piaget in 
structuring the mind. Computer 1.st-processing techniques are used in 
structuring and manipulating the memory for this task. The reason for 
this is that these techniques model the theoretical structures of the 
brain and are efficient in coding, requirements. 

1. The Memory Structure 

The memory structure of the program, depicted in Appendix A, 
is divided into two segments, the "temporary" or short-term memory and 
the "permanent" or long-term memory. The short-term memory contains 
data regarding the current game and the long-term memory contains the 
data which comprises the experience gained from previous games. 

The short-term sexory 1 used to maintain data necessary for 
determining the pattern of play being used by both the human player ard 
the computer. The information within this segment of memory grows as 
the game progresses. At the end of the game, pertinent data is summarized 
and transferred to the long-term memory, and the short-term memory is 
deleted or "forgotten." The temporary memory consists of arrays and 
lists for determining the reliability estimates used in the reasoning 
processes of the system. In addition, this segment of memory contains 
& pattern of strategies used by each participant. It also maintains a 
total of the number of times each strategy is selected. The former is 


used in generally classifying the methed of play which is being employed, 
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and the latter is used in determining the overall aggressiveness being 
exhibited. 

The long-*erm memory, the "experience," consists of arrays and 
lists arranged into four libraries. Libraries are interconnected as 
they are developed, in order to maintain continuity in the flow of in- 
formation and facilitate access to specific data. These four libraries 
are labeled the "Catalogue of Players," the "Library of Initial Moves," 
the "Library of Types of Moves," and the "Library of Sequences of Moves." 
For simplicity these names are abbreviated the "Catalogue," the ‘Initial 
Library," the "Type Library," and the "Sequence Library." 

To provide a basic understanding of the construction and main- 
tenance of these libraries, it is necessary to define various terms associ- 
ated with them. A move is a played strategy. Groups of three moves are 
categorized into five levels of aggressiveness. Groups are thus combina- 
tions of any three strategies played. The following example illustrates 
the construction of groups: | 

Suppose an opponent picked the following moves (strategies): 

bee? de BT 2 2 3s ea ee dS 
Group (1) then consists of 1 2 1, Group (2) is 2 1 1, G. wp (3) is 
1 13, Group (4) is 13 2, and so on. 

Patterns are determined in the Initial Library according to 
individual moves, and are determined in the Type Library by a sequence 
of groups. The sequence of groups is called a "TYPE."' Further dis- 
cussion will be given to the utilizatic. of categories of aggressive- 
ness, and their associated groups, in the explanation of the learning 


precess of the program. 
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The Catalogue contains information on one to ten players. As 
a player is first introduced into the game, a unique set of nodes (sequen- 
tial set of computer worde) is established for him which at the end of 
the game will be filled with data and pointers regarding his history of 
play. The most recent player is maintained at the top of the catalogue 
and the eldest player at the bottom. As players participate in addi- 
tional games, a reordering of the Catalogue is accomplished to maintain 
this perspective. If the Catalogue reaches its capacity of ter players 
and a new player is introduced into the system, the information on the 
oldest player is ‘eleted (forgotten) and the new player established in 
the Catalogue. 

Information maintained on each player includes pointers to the 
initial strategies and a sequence of "TYPE" patterns used by both the 
player and the computer in previous games. It also contains the set of 
parameters (the foundation of the coefficients considered in die stratery 
polynomial) which have proven most successful against that player, the 
average values of reliability demonstrated by both the player and the 
computer when playing against that player, and the player's aggressive- 
‘ness in previous games. | 

The INIT Library contains the initial three moves (first group) 
used in various games in the pasc, and pointers to the Type Library. 
Information on the exact initial strategies used in previous games is 
maintained so that the computer can try to get a "jump" on the player at 
the beginning of the game. It is used to predict SOUTH's initial strategy 
and to aid the computer in selecting the best offense against this 


strategy. 
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The Type Library contains up to ten structures. A structure 
consists of a header cell followed by a TYPE of eight to ten groups. 

The header cell contains pointers to other structures in the Type 
Library which have demonstrated a successf"1 defense against this TYPE, 
and against which this TYPE has proven to be a good offense. 

The Sequence Library is a history of the TYPEs and associated 
initial strategies for each game played by a particular player in the 
Catalogue. It also contains the computer's TYPEs used against any player 
in the Catalogue. Each entry in the Sequence Library contains a pointer 
to a structure in the Tvpe Library ani a pointer to the initial strategy 
associated with that struc.ure, 

When the capacity of each of these libraries is filled, the 
oldest information is deleted, or updated, to make room for the new in- 
formation. This conforms to the contention that the brain grows to a 
finite size, then information tecomes restructured or forgotten as it 
grows out of date. 

2. The Learning Process 

The learning process progresses as the memory structures 
develop. During the first game the computer plays, there is nothing in 
long-term memory so the computer must reason through the game as best 
it can. During succeeding games, however, the computer searches long- 
term memory for strategy predictions and pattern-matching types of 
strategics. 

When the player signs into the system at the start of a game, 


the computer scarches the Catalogue to determine if this player has 


been played before. If so, the computer assumes that he will fullow 


the same initial strategy and determines the best moves against that 
Strategy. After two moves the computer pattern-matches these against 
the INIT Library to find the closest match. The computer's third move 
is selected as the one which is best against the third move listed in 
the closest matched INIT group. After the third move, the computer 
again pattern-matches the initial three moves against the INIT Library, 
finds the closest match, and predicts that the player will use the TYPE 
to which that entry in the INIT Library points. 

The Type Libary provides general categories of strategies to 
simplify pattern-matching. In a twenty-move game, there could be an 
almost infinite number of sequences of exact strategies used, but by 
categorizing exact strategies into general classifications, pattern-match- 
ing may be done at a meta-level. These categories, as stated earlier, 
are determined by arranging the moves into groups of three and classi- 
fying the group into one of five levels of aggressiveness. Figures 6 
and 7 illustrate the techniques of categorizing strategies and making 
predictions based on the types obtained. The overlap of the groups pro- 
vides continuity in classifying and predicting. 

As the game progresses, the computer pattern-matches the TYPEs 
used by both the piayer and the computer, which are maintained in short- 
term memory, against the Type Library. This provides NORTH with pre- 
dicted moves for both participants, and also with an indication of the 
TYPE that is best tc use against SOUTH's TYPE. 

The Sequence Library provides the computer with the ability 


to make better predictions of the TYPEs that a player will use. After 


~o each game with a player, a pointer to the TYPE employed in that game is 


CATEGORY 1 Al 2 A2 3 


Combination of “111 113 123 133 332 
Groups of 
Three Moves 112 131 132 331 323 
121 311 213 313 233 
211 122 231 322 333 
212 312 232 
221 321 223 
222 


The sum of the strategies for each category is: 
Category 1 - 3 or 4 Category Al - 5 
Category 2 - 6 Category A2 - 7 


Category 3 - 8 or 9 


Figure 6. Category Values of Strategies Used in 
Pattern-Matching 


By totaling the first two moves of any predicted category, it is 


possible to predict the next move: 


If First For The Move 
En Two Moves Category Predicted 
12 | 2 | Total: Number: Is: 
Ebon fei || 1 2 
aaae ;  & 4 
2 1 Al 2 Al 3 
3 Al 2 
Overlapping of 4 Al 1 
Category Groups 
Shown Above 3 2 3 
4 2 2 
Example Strategies 5 2 1 
and Categories are 
Inserted 4 A2 3 
5 A2 2 
6 A2 1 
5 3 ) 
6 3 2 


Figure 7. Prediction of the Next Move Based on 
Strategy Type 
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inserted in the Sequence Library. The computer can pattern-match these 
lists to determine the expected type which the player will use during 
the next game. 

At the end of a game, corresponding data between the short- 
term memory and the long-term memory are compared. This is accomplished 
by the same methods of pattern-matching used during the game. If similar- 
ity exists, the structures of the shore term memory are combined with the 
closest Initial and Type Library structures to form more up-to-date in- 
formation. If no similarities exist, the information from temporary 
mewory is transferred to permanent memory as new Initial and Type Library 
entries. If these libraries es at capacity, the oldest information is 
deleted (forgotten) and the new information is inserted. 

As suggested by the theory, past experience is utilized to 
obtain the best prediction of events, but if the computer finds no match- 
ing experience upon which to draw, it must reason through the problem as 
best it can. Knowledge grows when new structures are formed or old ones 


reconstructed. 


C. ANALYSIS OF THE GAME 

After a rather poor start, the computer has gone ahead of ite op- 
ponents in total points for all games, and the gap is widening. This 
is due partly to some minor changes in the strategy decision routines, 
however evidence indicates that most of the credit can be given to the 
building of the libraries. This contention is supported by the fact 
that the analysis portion of the strategy routine tends to increase 
(tune up) the basis of the coefficient assigned to the experience factor, 


and tune down some of the others. 
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Based on observation of the game and discussions with those playing 
it, it is evident that the players and the computer use many of the same 
factors in deciding upon a strategy and a concession point. In addition, 
concepts built by the human are similar to the concepts formed by the 
computer and stored within its memory. The average values for the re- 
liability estimates that the computer maintains on both itself and its 
opponent remain relatively constant as the game progresses. The same is 


true of the weighted reliability. 
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Iv. CONCLUSIONS 


“Tf a program can be constructed that, when placed 

in a prisoner's dilemma situation, exhibits behavior 

like the behavior of people when placed in the same 

situation, then that program will be a powerful tool 

for generalizations." 

The above statement by Professor Weil is repeated from an earlier 
section of this thesis for emphasis. In its existing form, this program 
does provide a powerful tool for extracting generalizations regarding 
human behavior in a medium risk decision making task. With but minor 
changes in wording or by incorporating extensions to the game, the pro- 
gram can be made applicable to almost any field of corporate or govern- 
mental endeavor requiring a psychological understanding of human be- 


havior in making decisions where different gains can be achieved at 


varying risks. 


A. EXTENSIONS AND CHANGES 

DIPLOMAT is in itself an extension to the classic prisoner's 
dilemma game. It may be extended or changed to broaden its applicabil- 
ity to real world situations and make the game considerably more inter- 
esting. 

The simplest change to the program would be to change the name 
assigned to the three strategies. For example, the applicability of 
the program could be changed by transforming the words arm/status-quo/ 
disarm into the words buy/wait/sell, or perhaps raise-prices/no-change/ 
lower-prices. 


The whole outcome of the game can be changed drastically by changing 
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the POINTS formula. This is done by simply assigning different co- 
efficients to the £ ctors of WEALTH and STRENGTH, or by changing the 
values in Table I (the wealth changes for the various strategies shown 
as a function of player strength). Thus the game may be altered to 
match actual conditions encountered in, say, the business community. 

The number of strategies from which to choose could be changed in 
either direction. Decreasing the choice to two would render the game 
closer to the prisoner's dilemma situation, but even this hag many ap- 
plications in the real world. On the other hand, it would be more in- 
teresting to increase the number of strategies. For example, there 
could be two levels of arming and two levels of disarming, or for buying 
or selling. Some implications of increasing the number of strategies, 
however, world be that the pattern-matching routines may not be feasible 
in their present state. It would probably require a pattern-matching 
scheme which placed more emphasis on the meta-level, i.e., looking at the 
broad spectrum of the pattern from a higher level, rather than pattern- 
matching individual strategies or small groups of strategies. 

Increasing the number of players is probably the most difficult of 
the possible extensions to the game, but provides the most interesting 
possibilities. If the number of play:rs is increased, treaties Satieen 
the players and alliences among groups of players may be proposed end 
formed. An infinite number of situations may arise out of this idea, for 
example, one nation might wonder whether his ally will abide by the al- 
liance or possibly turn on him several moves hence; or, a player might 
hesitate to sign an agreement when a better one might be offered from a 
different player. In the diplomacy situation, several nations may dis- 


arm to gain wealth, then form an alliance against a stronger nation which 
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has armed to gain strength at the cost of wealth. Increasing the number 
of players would also make the cenespts of reasoning and learning more 
difficult, but more fascinating. For example, will one nation risk an 
alliance with another when it remembers prior treachery? Instead of 
merely analyzing what one's opponent thinks of him, a player will have 
to analyze several players’ estimates of all the opponents. The reli- 


ability considerations become almost overwhelming. 


B. POSSIBLE USES OF DIPLOMAT 

It is considered that Professor Weil was correct in assuming that 
the prisoner's dilemma computer game would be a powerful tool for 
generalizations, The guste of DIPLOMAT or its extensions would be an in- 
valuable aid in the training of executives prior to stepping into posi- 
tions requiring the art and finesse of personal contact. The speed of 
the computer permits the game to proceed rapidly and permits many differ- 
ent situations to be established by changing payoffs and formulas. The 
performance of different players against standard setups could then be 
snalyzed. 

In addition, personnel in the study of behavioral science and psy- 
chology could develop a better understanding of the nature of the human 
decision-making process and of the risk of striking out on an independent 
path as opposed to the benefits and security gained by cooperating. The 
fact that the risky path may lead to greater gains may be more important 
to some people than to others. 

The training of college students in the theory of marketing and 
analysis would also be enhanced by applying textbook concepts to the 


difficulties of the everchanging conditions and types of people with 


which they may deal. This training might prove useful for young officers 
in the diplomatic corps or even at junior level armed forces colleges. 

In order to enhance the development of computer application to this 
type of work, it is recommended that consideration be given to the 
possibility of adapting this program to marketing or financial decision- 
making cours:s in the management curriculum. This type of program could 
be extremely useful as a toot in teaching management students the power 
and usefulness of the computer. It demonstrates the interface capabil- 
ities of man and machine. It can also be used as a teaching aid in 
artificial intelligence, game theory, and basic management-decision 
courses. The first part of Section III describing the game could be 
reproduced for use as a handout for potential players. 

The concept and performance of DIPLOMAT appears to be good. It is 
ahead of the human players in POINTS and performs well in adapting to 
different human strategies that have been tried against it. However, 


similar to a human, it does not win every game, indicating that ev.n it 


has more to learn. 
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SUBROUTINE MOVESL 


SUBROUTINE TO PUT OLO PLAYER TO THE TOP OF THE CATALOGUF. 
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SUBROUTINE SEARCH(PLY) 


TO FIND NUT IF A PLAYER IS IN THE CATALQGUE 
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